Method and apparatus for determining volumetric data of a predetermined anatomical feature

ABSTRACT

A method of determining volumetric data of a predetermined anatomical feature is described. The method comprising determining volumetric data of one or more anatomical features present in a field of view of a depth sensing camera apparatus, identifying a predetermined anatomical feature as being present in the field of view of the depth sensing camera apparatus, associating the volumetric data of one of the one or more anatomical features with the identified predetermined anatomical feature, and outputting the volumetric data of the predetermined anatomical feature. An apparatus is also described.

This application claims the benefit of U.S. patent application Ser. No.15/757,364, filed Mar. 4, 2018, which is hereby incorporated byreference in its entirety.

FIELD OF THE DISCLOSURE

The invention relates to a method and an apparatus for determiningvolumetric data of a predetermined anatomical feature. A non-invasivemethod and apparatus for monitoring the volume of body parts, or partthereof, of an animal or human is presented, by the application ofmachine vision techniques to create and compare 3-dimensional (3D)models of the body parts over time. The invention has applicability inthe areas of human and animal health, particularly in monitoring theperipheral edema associated with a range of pathologies including heartfailure.

BACKGROUND

Water retention resulting from chronic heart failure can be detected bya number of methods, most commonly by measurement a weight gain or anincrease in limb volume.

The WHARF study by Goldberg et al. (Am Heart J. Volume 146(4), pages 705to 712, 2003) demonstrated that compliant patients monitoring theirweight on a regular basis could reduce the mortality from chronic heartfailure by 50%. Unfortunately, such benefits are not commonly observedin normal patient populations due to non-compliance to a daily regime ofweight monitoring, even when internet-connected weighing scales are usedto remove any necessity for patients to record or report the informationcollected.

Brijker et al. (Clinical Physiology, volume 20, issue 1, pages 56 to 61,January 2000) demonstrated that limb volume is a much more sensitivemeasurement for the purposes of heart failure monitoring than weightgain. Weight measurement has a number of known fluctuations, such aslevel of hydration and how recently one's bowel has been evacuated thatcan interfere with the desired signal indicative of excess waterretention. The study of Brijker et al. demonstrated that patients showeda change of approximately 5.9% in weight between hospital admission anddischarge, compared to a change of 13.1% in leg volume and a change of7.1% in leg circumference. In addition, the coefficient of variationbetween weight and leg volume measurement was only r=0.37, suggestingthat the two methods are somewhat independent, therefore leg volumemonitoring may add significantly to the assessment of edema compared tothe measurement of weight alone.

Unfortunately, the clinical gold-standard for limb volume measurement isa water displacement technique that suffers significant inter-operatorvariation and is cumbersome and error-prone to administer. Essentially,the limb of interest is immersed into a container of water to variousdepths, and the water displaced from the container is captured andrecorded.

Because patients are capable of surviving for many years following theonset of chronic heart failure, the problem of finding a solution to theroutine monitoring of changes in limb volume for the casually compliantpatient in a home setting is an area of activity.

Therefore, a multitude of devices have been devised to measure changesin limb volume, typically in the form of various “smart” socks orsimilar, as is described in U.S. Pat. No. 8,827,930B2. The fundamentalissue with such approaches is that patient non-compliance is asignificant factor in effective long-term monitoring.

An alternative, non-invasive approach is to apply “machine vision”concepts, wherein the limb in question is measured using a 3D imagingsystem. Hayn et al. applied the Microsoft Kinect depth-measuring camerasystem to the measurement of leg volume for the detection of edema(ATS13091B1). The approach taken by Hayn et al. was to identify aspecific set of reference points, lines, curves or planes that had beenshown to correlate with limb volume. Unfortunately, this approachprovided only modest correlation with weight, and also requires that thepatient is oriented in a particular way with respect to the imagingsystem in order to capture the reference points/lines/curves/planes.This is clearly susceptible to the same compliance issues noted above.

Other approaches to 3D imaging of limbs have been taken, typically inthe hospital setting which involves the use of a 3D imaging system forroutine measurement of limb volume. Unfortunately, such systemstypically require the patient to be rotated with respect to the imagingsystem, and to have limbs not of interest to be covered so as not tointerfere with the observation of the limb of interest. This is clearlynot practical for home use.

An alternative approach is described in EP0760622A1, which describes aparticularly inexpensive 3D scanner, in which a body part to bedigitized is provided with an elastic cover that carries marks to beevaluated by photogrammetry. Overlapping images are taken using one ormore roughly positioned cameras, and a 3D model of the body part isgenerated from the combination of these 2-dimensional (2D) images.Unfortunately, such a system is impractical for home-use by the casuallycompliant patient as they cannot be expected to diligently wear specialclothing.

Stocking-based systems have been the subject of significant academicresearch, for example Hirai et al. in “Improvement of athree-dimensional measurement system for the evaluation of foot edema”(Skin Research and Technology, volume 18, issue 1, pages 120 to 124) andPratsch et al. in “Perceived swelling, clinical swelling and manifestlymphoedema-difficulties with the quantifying of leg swellings”(Phlebologie, volume 41, page 5 to 11, 2012). However, the issue withusability remains significant.

Other methods and devices relating to image analysis are described inU.S. Pat. No. 8,908,928B1, US2011295112A1 and US2015216477A1.

Accordingly, an aim of the present invention is to provide anon-invasive, opportunistic method and apparatus for monitoring theperipheral edema associated with a range of pathologies including heartfailure in animals or humans, and reporting the results thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure can be understood with reference to thedescription of the embodiments set out below, in conjunction with theappended drawings in which:

FIG. 1 illustrates an apparatus according to an embodiment of theinvention;

FIG. 2 illustrates 3D image data of a foot as it is observed by a depthsensing camera apparatus;

FIG. 3 illustrates an example of a reference 3D model of a foot;

FIG. 4 illustrates a 3D model after adjustment of a parameter for aleg-to-foot-angle has been performed;

FIG. 5 illustrates the 3D model after adjustment of a second parameterfor lower-leg-conical-volume.

FIG. 6 illustrates the resulting model which more closely matchesobserved 3D data;

FIG. 7 illustrates an image of a foot captured by a 3D imagingapparatus;

FIG. 8 illustrates the image of the foot rotated about an axis, showingthat the other side of the foot has not been imaged;

FIG. 9 illustrates a 3D model to be fitted to the 3D image dataillustrated in FIG. 7;

FIG. 10 illustrates the 3D model after it has been fitted to thecaptured 3D image data; and

FIG. 11 illustrates a method according to an embodiment of theinvention.

DETAILED DESCRIPTION

According to a first aspect of the invention there is provided a methodof determining volumetric data of a predetermined anatomical feature,the method comprising: determining volumetric data of one or moreanatomical features present in a field of view of a depth sensing cameraapparatus, identifying a predetermined anatomical feature as beingpresent in the field of view of the depth sensing camera apparatus,associating the volumetric data of one of the one or more anatomicalfeatures with the identified predetermined anatomical feature, andoutputting the volumetric data of the predetermined anatomical feature.

The step of identifying a predetermined anatomical feature as beingpresent in a field of view of the depth sensing camera apparatus may beperformed before the step of determining volumetric data of one or moreanatomical features present in the field of view of the depth sensingcamera apparatus.

Upon identification of the predetermined anatomical feature beingpresent in the field of view of the depth camera apparatus, the methodmay comprise obtaining three-dimensional data of the predeterminedanatomical feature and determining the volumetric data of thepredetermined anatomical feature based on the three-dimensional data.

The method may comprise acquiring a two-dimensional image from the depthsensing camera apparatus and identifying the predetermined anatomicalfeature as being present in a field of view of the depth sensing cameraapparatus in accordance with the two-dimensional image.

The step of identifying a predetermined anatomical feature as beingpresent in a field of view of the depth sensing camera apparatus may beperformed after the step of determining volumetric data of one or moreanatomical features present in the field of view of the depth sensingcamera apparatus.

The method may comprise obtaining three-dimensional data of the one ormore anatomical features and determining the volumetric data of the oneor more anatomical features based on the three-dimensional data.

The volumetric data of one of the one or more features may be associatedwith the identified predetermined anatomical feature by comparing thevolumetric data of each of the one or more anatomical features with athree-dimensional representation of a predetermined anatomical feature,and identifying one of the one or more anatomical features as being thepredetermined anatomical feature.

The step of determining the volumetric data may comprise processing thethree-dimensional data in dependence on a three-dimensionalrepresentation of the predetermined anatomical feature.

The three-dimensional representation of the predetermined anatomicalfeature may be a deformable three-dimensional model, and wherein themethod may comprise parametrically deforming the three-dimensional modelto fit the three-dimensional data of the predetermined anatomicalfeature.

The three-dimensional model may be deformed by one or more of rotation,altering a joint angle and volumetric change.

The three-dimensional representation of the predetermined anatomicalfeature may be a deformable three-dimensional model, and wherein themethod may comprise deforming the three-dimensional data of thepredetermined anatomical feature to fit the three-dimensional model.

The three-dimensional data may comprise multiple data sets of anatomicalfeatures at different orientations.

The data may be periodically or continuously received from the depthsensing camera apparatus.

The three-dimensional data may comprise three-dimensional image data.

The three-dimensional data may represent at least one partialrepresentation of the predetermined anatomical feature.

The predetermined anatomical feature may be a limb, and may be a foot.

The depth sensing camera apparatus may comprise at least one emitter andone detector array, or at least two detector arrays.

The method may comprise recording the time at which three-dimensionaldata is captured, and wherein the volumetric data may be generated independence on the recorded time.

According to a second aspect of the invention there is provided a methodof determining volumetric data of a predetermined anatomical feature,the method comprising: identifying a predetermined anatomical feature asbeing present in the field of view of the depth sensing cameraapparatus, determining volumetric data of one or more anatomicalfeatures present in a field of view of a depth sensing camera apparatus,associating the volumetric data of one of the one or more anatomicalfeatures with the identified predetermined anatomical feature, andoutputting the volumetric data of the predetermined anatomical feature.

According to a third aspect of the invention there is provided a systemcomprising: a depth sensing camera apparatus; and a processor coupled tothe depth sensing camera apparatus; wherein the processor is configuredto perform any of the methods described above. The depth sensing cameraapparatus may comprise at least one emitter and one detector array, orat least two detector arrays.

According to a fourth aspect of the invention there is provided acomputer readable medium having stored thereon instructions which, whenexecuted on a processor, cause the processor to perform any of themethods described above.

FIG. 1 illustrates an apparatus 100 according to an embodiment of theinvention. The apparatus 100 or system includes a depth sensing cameraapparatus. The camera apparatus in this embodiment includes twoinfra-red and visible light sensitive cameras 102, 104. Each camera 102,104 also includes a controlled infra-red (IR) emitter. Accordingly, theapparatus 100 may continuously observe an imaging volume or field ofview without disturbing a patient. In the preferred embodiment, thedepth sensing cameras are both IR and visible light sensitive, such thatcontrolling the IR emitter allows the imaging system to capture at leastIR images and 3D data. If visible lighting is sufficient, the camera mayalso be used to capture visible color information. This arrangement hasthe benefit of not disturbing the patient, whilst capturing colorinformation when possible. Color information may be a useful adjunct tolimb volume information when a medical professional is assessing apossible edema.

It will be appreciated that the camera apparatus may include at leastone infra-red sensitive camera, and at least one controlled lightprojector (e.g. infra-red emitter or laser), or at least two spacedapart conventional cameras (i.e. cameras containing a single 2D imagingarray.

The apparatus 100 further includes a processing device 106 whichreceives image data, in this example 2D image data, from the camera 102.The processing device 106 is configured to identify if a limb ofinterest (e.g. foot 200) is present within the 3D environment imaged bythe camera 102, for example. The processing device 106 includes 2D imageanalysis software or hardware, which is capable of identifying apredetermined limb from a 2D image. The analysis software or hardwareutilizes similar algorithms to those used for face recognition. Forexample, elements of the limb are identified by extracting landmarks, orfeatures, from a previously acquired image of the patient's limb. Analgorithm is consequently used to analyze the relative position, size,and/or shape of elements of the limb (e.g. toes, ankle and heel of afoot 200), and these elements are used to search a 2D image. Radiotagging of the limb of interest may also be used to more easily identifyif the limb of interest is in the field of view or imaging environmentof the camera 102. The processing device 106 is configured to triggerthe acquisition of 3D image data from the cameras 102, 104, when thepatient's limb is identified as being present in the field of view ofthe camera 102. The 3D image data are acquired by a processing device108.

The processing device 108 acquires the image data from the depth sensingcamera apparatus 102, 104, and is configured to extract volumetricinformation about the patient's limb. The volumetric data issubsequently passed or transferred to a device 110, for reporting. Thedevice 110 is a personal computing device and includes a user interface(e.g. screen/monitor and an input device) for allowing a user tovisualize the volumetric data. The device 110 in this example is locatedin the same location as the depth sensing camera apparatus, and isconfigured to communicate over the Internet, including a GSM network, oran intranet, for example, with a remotely located computing system 112or cloud computing system. The processing device 108, may alsocommunicate directly with the remotely located computing system 112,such that computing system 110 is not required. The remotely locatedcomputing system 112 may be accessible by a medical practitioner orother care worker.

In this example each of the processing devices 106, 108 includes aprocessor capable of launching and running software programs (e.g.,applications) stored in a memory (e.g., RAM and/or ROM), and aninput/output interface configured to communicate with each other, thedepth sensing camera system 102, 104 and the computing device 110 and/orthe computing system 112. It will be appreciated that the two processingdevices 106, 108 could be provided as a single device, and may includeone or more applications or software programs, or may include one ormore application specific integrated circuits. In this regard, theprocessing devices 106, 108, may be packaged together with the depthsensing camera system 102, 104 in a single vendible unit, which isconfigured to communicate with the computing device 110 or the remotelylocated computing system 112. Such a single vendible unit may alsoinclude a user interface (e.g. a touch sensitive display/touch-sensitiveoverlay on a display). Each of the depth sensing camera system andprocessing devices 106, 108, 110 may also include a short-range,wireless communication system and may include a wireless bus protocolcompliant communication mechanism such as a Bluetooth® communicationmodule to provide for communication with similarly-enabled systems anddevices or IEEE 802.11 radio standards, which is more typically known asWiFi.

The method which is performed by the apparatus 100 illustrated in FIG. 1is now described in associated with FIGS. 2 to 10.

The depth sensing camera apparatus 102, 104 is arranged within the homeenvironment of a patient, in such a manner that opportunistic images ofthe feet/legs are likely to be observed. It has been found that locatingthe depth sensing camera apparatus 102, 104 near the bed to scan thearea from which the patient is likely to get into bed and/or get out ofbed, is particularly advantageous as there are few other times whenpatients will typically have exposed feet/legs without footwear, socksetc. Furthermore, it has been found that a horizontal arrangement ofcameras inside a bar-like enclosure is suitable for attachment to a wallat a distance of approximately 70 cm (i.e. a distance in the range of 50cm to 100 cm) from the floor and 1 m from the bed, with a distancebetween cameras of 100 mm in this example. Other arrangements andpositions of cameras are contemplated. For example, near an access pointto a shower or bath area. Alternative arrangements include, for example,a semi-circle of cameras around an imaging volume. Although preferablefor image collection such a semi-circular arrangement may be consideredto be more obtrusive in the patient's environment.

The depth sensing camera system optionally includes the facility toautomatically adjust various lighting conditions, without affecting thepatient. Preferably this is performed through the use of color cameraswithout infra-red filters, such that in low-light conditions an infraredLED can be used to provide IR light. The depth sensing camera system isconfigured to dynamically adjust the IR LED power to provide optimallighting conditions for monitoring the scene, as is known in the art.

The processing device 106 is configured to receive image data formcamera 102 and to monitor the environment or field of view of the camera102 for the object of interest (i.e. a patient's foot 200). This isachieved continuously, by capture and video analysis, or periodically,e.g. once per second. Video analysis (i.e. continuously) is preferred,since the capture of 3D image data can be triggered when the feature orobject recognition has determined that a foot is present in the field ofview or environment of the depth sensing camera apparatus, and the footis determined not to be moving too fast (e.g. less than 100 mm/s)through the environment viewed by the depth sensing camera apparatus. Ifthe foot is moving faster than 100 mm/s, motion blur may be introducedinto the 2D images and depth artefacts may be introduced into the 3Dimage data. The speed of the foot may be determined by measurement ofthe distance the foot has moved in between video frames, for example.However, video speed recognition may impact limitations on the amount ofprocessing that can be performed for the foot recognition. Therefore, itis preferable to opportunistically trigger the collection of 3D imagedata on the basis of a fast estimate that a foot may be present, andlater discard any data image if it is determined that the foot ofinterest is not in fact present. Once a foot has been optimisticallyrecognized, and optionally determined to be moving slowly enough tocollect robust 3D image data, a 3D measurement is triggered. It is alsopreferable to match an imaged foot against a set of target feet (i.e.multiple matches) to minimize false-triggering on feet that do notbelong to the patient.

The 3D measurement includes collecting or acquiring at processing device108 images from the cameras 102, 104 of the depth sensing apparatus 100,which are focused on the imaging volume or field of view. In thisexample two cameras are used but from different locations. Thetrigonometric relationship between the scene observed or environmentobserved by each camera (or projected by each projector) enables acalculation of the distance from each camera or projector. Thisrelationship allows one to calculate the maximum depth resolution thatis possible for a given set of cameras and projectors, which is afunction of the separation between the cameras/projectors (which in thisexample is 100 mm), the separation between the cameras/projectors andthe target being measured (which in this example is 1 m), and theangular resolution of each pixel in the cameras/projectors, which is afunction of the camera/projector optics and the physical size of eachpixel. Software used for such 3D analysis is commonly referred to as “3Dreconstruction”, “Multi-View Stereo” or “Structure from Motion”software, which typically either matches pixels betweencameras/projectors to work out the 3D distance to the target, or buildsa 3D simulation of the target that best matches the observations made(i.e. either solving a matching problem or optimizing the inverseproblem). This analysis may be further refined using techniques of“Structure from Shading” where the detail of a surface can bereconstructed from 2D images collected given some assumptions about thelighting present in the scene.

The 3D data or measurement comprises multiple, preferablyhigh-resolution, images taken in succession from the sensor array ofeach of the cameras 102, 104. The images from each camera are typicallysynchronized in time, to avoid artefacts due to motion within the scene.This is done to reduce the depth-uncertainty due to thetiming-uncertainty in triggering for a maximum rate of movement in thescene. For example, if two cameras 100 mm apart with angular resolutionof 0.02 degrees per pixel are configured to monitor a scene 1000 mm awayfrom the cameras, the depth (z) resolution will be of the order of 5 mmfor a 1-pixel uncertainty between cameras. The (x,y) resolution of eachpixel in each camera is therefore of the order of 0.5 mm. Accordingly, a0.5 mm uncertainty in position across the image due to movement combinedwith camera synchronization error will correspond to approximately a 5mm error in depth perception. If a foot during a slow walk, whilststanding on the floor but having a foot-to-leg angle change, is assumedto move at 100 mm/sec, then timing jitter should be kept below 5 msec(i.e. 0.5/100), and preferably much less than this. This can be achievedusing electronically synchronized systems.

It is preferable to assist later stereoscopic matching algorithms, ifthe depth sensing camera apparatus does not contain a known lightprojector, by at least configuring the apparatus to control a “textureprojecting” light source, which projects a pattern of IR light over thescene so that objects of uniform color have at least some textureprojected onto them. It is also preferable, if the camera isIR-sensitive, to take measurements at multiple levels of IR lightingsuch that extrapolation to zero IR lighting is possible in order toextract visible color information from the camera more reliably. Thisvisible color information may be useful to medical professionals invisually assessing a possible edema that the system identifies. Colorinformation is also useful in the assessment of wound healing, for which3D and color information is a particularly useful combination.Therefore, the 3D image capture process preferably includes a series ofsynchronized images taken from multiple cameras with various levels ofIR illumination and IR texture projection applied.

Once 3D image data has been collected by processing device 108, the datais processed by a quality-estimation algorithm at the processing device108 to determine the likelihood of good quality data resulting from thesubsequent processing stages. If it is determined that the image data islikely to give rise to a good result, the image data are entered in to aqueue in the memory of processing device 108 for processing, with thedata having the highest anticipated good result first. For example, ananticipated good result will be expected from an image set where thefoot is approximately centrally located in the imaging volume, is movingslowly, is not occluded by other objects in the scene, and the footorientation is not very similar (i.e. is dissimilar) to other recentlyacquired samples. This is beneficial, since the applicant has found thatonce a patient is observed by the system once, it is more likely thatthe patient will be observed multiple times within a short period oftime (e.g. they are walking around a room). The system then allows forprioritization of the “best” data, as well as data from the greatestrange of orientations observed, if there is limited storage space orprocessing time available within the system, and in cases of resourceconstraints (e.g. storage space or time) can allow for expected poorimage sets to be deleted prior to full processing. The use of a queueensures that the combination of temporary storage space and processingtime is optimally used, with the queue generally never emptyingcompletely (i.e. the predicted lowest-quality data never actually getsprocessed, it is discarded by the arrival of newer high-quality datathat causes the queue to overflow). This queue system located in thememory of the processing device 108 can be expanded to multiple queuesif, for example, while the full quality-estimation is performed, moreimage data may be acquired. For example a first queue might be based on2D image quality and positive recognition of the patient's foot ratherthan those of other occupants of a house, and a second prioritizationqueue might be based on incorporating 3D processing to help prioritize awide range of orientations of the foot to the imaging system.

Image data are retrieved from the top of the queue and a point cloud(i.e. 3D data) is generated by the processing device 108 using a 3Dreconstruction algorithm, as is known the art. The 3D image data foreach acquisition or exposure is derived from at least two images, eitherfrom the two separate cameras, or from one projector and at least onecamera. A summary of several such algorithms is provided by D.Scharstein and R. Szeliski, “A taxonomy and evaluation of densetwo-frame stereo correspondence algorithms” (International Journal ofComputer Vision, 47(1/2/3), page 7 to 42, April-June 2002). Theprocessing device 108 also performs various image processing functionson the image data, for example, rectification, and also on the pointcloud 3D data, for example, noise removal. The point cloud 3D data maybe further processed to remove background artefacts. FIG. 2 illustratesan example of the 3D image data of a foot as it is observed by the depthsensing camera apparatus 102, 104.

The point cloud 3D data is then used in association with a 3D model of afoot in order to fit the 3D model to the point cloud 3D data. The limbof interest is identified using the same 2D recognition used to identifythe foot previously, or a separate 3D recognition method is used, as isknown in the art. FIG. 3 illustrates an example of a reference 3D modelof a foot. This is achieved by parametrically adjusting the 3D modeluntil a good match is found between the 3D model and the point cloud 3Ddata. For example, the well-known iterative closest point method may beused whereby a random selection of points from the point cloud 3D dataare matched to their closest points in the 3D model, whilst theparameters of the model are adjusted, to find the best set of modelparameters (e.g. orientation, joint angles, volumetric distortions andso on). This is repeated using different random samplings so that theimpact of outlier points is minimized, until a good match between the 3Dmodel and the observed data is found (i.e. this is repeated until amatch with the lowest error is found). For example, the original 3Dreference model may be oriented with the left side of the foot facingthe camera, a leg-to-foot angle of 90 degrees, and a lower legrepresented by a cylinder having a circumference of 30 cm and a heightfrom the floor of 25 cm. The best fit of the 3D model to the point cloud3D data, in this example, is with the foot facing the camera front on,with an 80 degree foot-to-leg angle and a 33 cm lower-leg circumference.If a good match cannot be achieved, the 3D image data is discarded, andthe process is started again using the next set of 3D image data fromthe queue. It will be apparent that at least two images are used togenerate the point cloud, and these two images representthree-dimensional image data, but more than two images may be used.

The 3D reference model may be modified to exclude regions that areparticularly variable or problematic to measure, such that measurementprecision of the modelled regions is improved. For example, the toes onthe foot consist of a large number of joints that may be at a range ofrelative orientations, as well as a number of regions between toes thatare difficult to measure effectively during any one observation.Therefore a reference 3D model of a foot may be truncated a certaindistance from the back of the foot, in order to exclude this high-noiseregion from calculations of foot volume for edema monitoring purposes.For example the front 5 cm of the foot may be cut from the 3D model toremove the toes and complex joints surrounding the toes, since fittingso many parameters may be challenging and not relevant when determiningthe changes in the volume of the foot.

FIG. 4 illustrates the 3D model after adjustment of a parameter forleg-to-foot-angle has been performed. FIG. 5 illustrates the 3D modelafter adjustment of a second parameter for lower-leg-conical-volume. Theresulting model is illustrated in FIG. 6 which more closely matches theobserved 3D data and allows a calculation of 3D volume data with respectto the reference model to be calculated. For example the change involume due to the lower-leg-conical-volume is determined without anyimpact from the change in leg-to-foot-angle, as is discussed below.

The distorted 3D model is subsequently modified by the processing device10 to reverse or dis-apply non-volumetric distortions which wereperformed when matching the 3D model to the point cloud 3D data. Forexample, the 80-degree foot-to-leg angle is transformed or reversed backto 90-degree foot-to-leg angle, but the lower leg circumference of the3D model which was matched to the point cloud 3D data is retained.Accordingly, volume-relevant changes are maintained, and the 3D model isotherwise returned to its reference state.

The processing device 108 integrates the volume-matched model over aspecified region, for example, at each 1 mm portion from the floor orbottom of the 3D model in the illustrated orientation. In this manner,the distortable model provides a consistent mechanism to obtain volumedata as though the foot were in a reference orientation, even allowingthe arbitrary orientations and joint angles that are observed whenopportunistic data is collected. Thus, volumetric data in the form of avolume of the observed limb, or feature is are obtained.

The source images (2D and 3D images) may be “masked” to blank out pixelsthat are not relevant for fitting with the foot model, for example,pixels that might correspond to other objects in the patient'senvironment. This is done for patient privacy should these images betransmitted to any care workers.

The resulting foot model parameters and calculated volume, as well asthe masked images, are then transmitted to a cloud service where alarmscan be triggered for care workers if a concerning trend in volume changeis observed. A care worker may then inspect the relevant images and/orfoot models in order to assess the severity and urgency of the case, andproceed with a care pathway. As describe above, the processing device108 may output the resulting data to a local computing device 110, whichsubsequently transmits the data to an external device 112 such as acloud service, using an Internet or intranet connection.

The method and apparatus have been described for a cycle of imagecapture and analysis. However, it will be appreciate that the process offitting the 3D model to the image data may be repeated periodically soas to generate a profile of volume data against time. This profile maybe transmitted to a practitioner or care worker. The process may beperformed periodically (e.g. once a day, week or month), or may beperformed every time that the capture of 3D image data is triggered. Forexample, the processing device 108 may include a pre-stored schedule,which may trigger the capture of images, or may trigger the imageprocessing based on opportunistically acquired images. The schedule maybe once a day, once a week, or once of month, or any other periodspecified by a practitioner. Therefore, the invention provides amonitoring and reporting system and method.

The generation of the initial 3D foot model upon which processing isbased is now described. This may be collected in a number of ways,either using a separate reference scanning system (e.g. 3D laserscanner, or MRI scanner), or by having the patient perform a series of3D image captures under known conditions using the same hardwaredescribed above. In either case, the resulting full limb 3D data, takenat multiple orientations and joint angles, is used to generate aparametrically distortable model for later processing. Theparametrically distortable model is designed such that it matches aswell as is practicable to the full limb 3D image data at multipleorientations and joint angles, without implying any change in theoverall volume of the foot. This may be achieved either by generating anaccurate multi-parameter model including all of the relevant joints,muscles, and so on, or by using a simpler parametric model and includinga set of “volume cancelling” terms within the model. For example, it hasbeen found that if the model systematically underestimates total volumeas the leg-to-foot angle increases, a “volume cancelling” term may beapplied to increase the apparent volume in accordance with theleg-to-foot angle observed.

It will be appreciated that over time, the patient's body may adjust inways that lead to the reference 3D model becoming systematicallyunrepresentative, for example, as a result of diet or exercise.Therefore, the initial reference 3D data at a wide range of orientationsand joint angles can be combined with the 3D data collected over time toyield a slowly self-adjusting reference 3D model. This may not besuitable for some application areas where subtle changes in absolutevolume over long time-periods are important, but for edema monitoringfor heart failure patients the change in volume that is clinicallyrelevant is of the order of 10% over two weeks. Therefore aself-adjusting model with a time-period of months should not negativelyimpact on edema assessment, whilst ensuring that the model remainsrelevant over extended periods of time and avoiding the need forperiodic re-calibration. Accordingly, it is preferable to update the 3Dmodel periodically when a new reference model becomes available, forexample as the result of a full scan of the limb in question.Alternatively, the 3D model is incrementally updated over time.

The use of a deformable 3D model as the reference to which captured datais compared during any one measurement is advantageous when applied totime-series collection of data in an opportunistic fashion, as in thepresent invention. Although any one collected set of 3D image data canbe mapped onto the 3D model, any one set of 3D image data is highlyunlikely to describe all of the changes between the 3D reference modeland the current state of the limb under consideration. For example, theimaging system may only capture 3D data about the side of the limbfacing the depth sensing camera apparatus, if the depth sensing cameraapparatus does not cover a full 360 degree circumference (or fullsolid-angle of a sphere). However, over time a series of suchopportunistically collected 3D image data is likely to describe all, orbroadly all, of the surfaces of the 3D reference model. The reference 3Dmodel can therefore be updated over time from this time-series. Forexample, as the patient moves around their environment, the limb underconsideration may be observed by the imaging system over a range oforientations, allowing the reference 3D model to be evolved over time.Preferably, regions of the 3D model are only updated using data thathave passed pre-defined quality and consistency checks, for examplehaving observed the same consistent discrepancy between the observeddata and the 3D reference model over at least a certain number ofobservations separated in time by at least a certain period. In thisway, for example, the “texture” of a 3D model might be automaticallyupdated after a patient gets a tattoo, and the system determines thatthis is not some temporary change in skin complexion.

It has been reported that limb volume may alter depending on recenthistory, e.g. if the patient has been sleeping horizontally just beforemeasurement, vs standing vertically. Therefore, the apparatus may alsorecord the time at which the images are captured and this temporal datamay be output along with the volumetric data/profile. Furthermore, thistemporal data may be used as an input to the 3D distortable model inorder to compensate for the time at which the images are captured. Forexample, if the apparatus regularly captures a user effectively gettingout of bed in the morning, vs getting into bed in the evening, theprocessing device 110 may construct a model of this typical variationover several days of measurements, or may use this temporal data tofurther distort the 3D distortable/deformable model to compensate forthe time of day at which the 3D images are captured.

FIGS. 7 to 10 illustrates an example in which volumetric information isobtained using the techniques described herein when only part of thelimb is visible to the depth sensing camera apparatus 102, 104. FIG. 7illustrates a foot as presented to the 3D imaging system. FIG. 8illustrates the image of the foot rotated about an axis, showing thatthe other side of the foot is completely missing in this view. FIG. 9illustrates a 3D mesh model to be fitted to the 3D image data. FIG. 10illustrates the 3D model as it has been fitted to the observed 3D imagedata. Thus, the extraction of full volumetric data from this 3D model ispermitted even though only part of the limb is visible to the imagingsystem.

FIG. 11 illustrates a method of determining volumetric data of apredetermined anatomical feature according to an embodiment of theinvention which may be performed by the processing device 108 or 110.Each of the steps may be performed in a different order (for example,step 304 may be performed before step 302), and may be omitted.

In step 302, volumetric data of one or more anatomical features presentin a field of view of a depth sensing camera apparatus are determined.

In step 304, a predetermined anatomical feature is identified as beingpresent in the field of view of the depth sensing camera apparatus.

In step 306, the volumetric data of one of the one or more anatomicalfeatures is associated with the identified predetermined anatomicalfeature.

In step 308, the volumetric data of the predetermined anatomical featureis output.

The apparatus 100 illustrated in FIG. 1 and described above, is capableof identifying not just a general limb of interest, but of identifyingwhich among a corpus of possible limbs is currently in the field of viewdepth sensing camera apparatus. Thus, the apparatus could classify, forexample, both the left and right feet of multiple occupants of a houseprovided that sufficient training data were available. In this way, theapparatus may trigger full 3D scanning only when one of the desired setof limbs of interest is within view. This also allows the apparatus totrack multiple patients and/or limbs per patient.

In order to obtain clinically relevant volume measurements for edemadetection, it is advantageous to obtain volumetric measurements with aprecision of 5%, over a volume of around 2.5 liters, it is found that adimensional accuracy of the order of 2.5 mm is required assuming equalerrors in each dimension. Therefore, the apparatus described herein iscapable of achieving at least 2.5 mm measurement resolution in eachdimension for each pixel, and preferably better than 1 mm resolution inall 3 dimensions.

In an alternative embodiment, the reference 3D model may remain fixed,and a series of distortions are applied to the observed, point cloud 3Ddata to match the data the reference 3D model. Where references are madeto distortions of the 3D model to match to the observed, the point cloud3D data may be distorted via the inverse process to match the data thereference 3D model. Likewise, a further alternative embodiment couldmake adjustments to the 2D image data collected such that it matches the3D model, or adjustments could be made to the 3D model and renderingtechniques could then be used to match the observed 2D image data.

The applicant has found that it is possible to use 3D imaging systemswith a resolution poorer than 2.5 mm throughout the scanning volume.However, this becomes increasingly reliant on the statisticaldistribution of multiple depth measurements at different pointsproviding a correct “average” depth measurement, and thus providing anaccurate match to a distortable 3D limb model. For example, fitting alow resolution point cloud to a higher resolution 3D model is possibleby calculating the least-squares error in positions between points andthe model. Adjusting a series of 3D model distortion parameters tominimize this least-squares error, assuming that each low-resolutionpoint has independent errors associated, may result in a more reliableset of 3D model distortion parameters. However, it is apparent that asthe measured points have increasingly poorer resolution, andparticularly if the points do not have independent measurements betweenthem, that this technique degrades rapidly. The applicant has found thatthis is the case, for example, when depth measurements are quantized asoften happens as the result of stereo block matching algorithms invarious categories of camera-based 3D scanners, or when many x,y pointsshare the same depth information. Therefore, the apparatus describedherein is configured to acquire image data with at least a 2.5 mmmeasurement resolution in at least two dimensions for each pixel,coupled with a model to statistically improve the resolution of thethird dimension to better than 2.5 mm after processing.

In combination with the technical implementation provided above,involving the use of a full 3D model of the limb of interest includingboth orientation changes between the limb and camera apparatus as wellas relative orientations between the joints in the limb, it becomespossible to use the “silhouette” technique to enable quantification oflimb volume with respect to a detailed 3D model, by applyingmathematical distortions to the model in order to match the observedhigh-resolution 2-dimensional imaging in dependence on thelower-resolution depth measurements. In this approach, the smallquantization errors in depth measurement are effectively ignored, as thedepth information is used to estimate the real-world area occupied byeach observed pixel, and thus the real-world distance between pixelsthat make up the silhouette of the limb. For example, a 100 mm widesilhouette of an object as viewed by the camera at a distance of 1000 mmand an uncertainty of +/−10 mm in depth, yields an uncertainty in widthof just +/−1 mm in the silhouette width. Therefore, provided that the 3Dmodel is appropriately oriented and distorted to match the observeddata, this silhouette data may overcome the depth-measuring errors to alarge extent.

The acquired data has been typically described as being image data.However, if a known light source (e.g. a laser) and sensor array areused, it will be appreciated that the data are a representation of thedistance from each pixel in the array to objections in the field of viewor environment of the sensor array.

In a further embodiment, 3D image data is collected without firstdetermining the presence of the feature (e.g. limb) of interest. This 3Dimage data is then processed as described previously to yield a 3D pointcloud, against which a distortable model of the feature of interest isiteratively adjusted to maximize fit. It will be appreciated that at nostage here was an explicit recognition of the feature of interestperformed, however in the case that the feature of interest is present,then the parameters of the distortable/deformable model will converge onmeaningful values. However, implicit recognition of the feature ofinterest is possible, for example by setting allowable limits onparameter values for the model, or threshold levels on the quality offit between the observed data and the 3D model. Therefore it will beunderstood that an implicit step of determining the presence or absenceof the feature of interest may be performed after all 3D data captureand model fitting is complete. This may take the form of filtering theresulting model parameters (e.g. foot volume measurements) from atime-series of 3D image data, and retaining only data from thosetime-points that meet some data quality criteria. It will also berecognized that, rather than using an absolute threshold on data qualityor the presence or absence of the feature of interest, it is possible toweight the results of the analysis by some measure of goodness-of-fit orcalculated probability that the observations correspond to the featureof interest. In this way, feature recognition can be both implicit andfuzzy rather than a simple binary cut-off, and the accumulated resultsof analysis will tend to represent predominantly information from thefeature of interest over multiple measurements.

A technique for obtaining the volume of a foot has been described.However, it will be appreciated that the technique may be applied toother limbs, or features (e.g. scars, wounds, etc.) of limbs, or anyother anatomical feature or limb. Moreover, the technique describedherein has focused on the use of measuring the volume of a foot, or feetfor targeting heart failure. However, it will be appreciated that thetechnique may be used during pregnancy or for lymphoedema associatedwith breast cancer.

The embodiments described in accordance with the present invention maybe provided as a computer software product. The computer softwareproduct may be provided in, on or supported by a computer readablemedium which could be provided as all possible permanent andnon-permanent forms of computer readable medium either transitory innature, such as in a data transmission signal for example sent over theinternet, or non-volatile storage such as memory. On the other hand thecomputer readable medium may be a non-transitory computer readablemedium comprising all computer-readable media.

The term “computer readable medium” (or non-transitory computer readablemedium) as used herein means any medium which can store instructions foruse by or execution by a computer or other computing device including,but not limited to, a portable computer diskette, a hard disk drive(HDD), a random access memory (RAM), a read-only memory (ROM), anerasable programmable-read-only memory (EPROM) or flash memory, anoptical disc such as a Compact Disc (CD), Digital Versatile Disc (DVD)or Blu-ray™ Disc, and a solid state storage device (e.g., NAND flash orsynchronous dynamic RAM (SDRAM)). It will be appreciated that theforegoing discussion relates to particular embodiments. However, inother embodiments, various aspects and examples may be combined.

While the invention is described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that the invention is not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit the invention tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present invention. The headings used herein arefor organizational purposes only and are not meant to be used to limitthe scope of the description. As used throughout this application, theword “may” is used in a permissive sense (i.e. meaning “might”) ratherthan the mandatory sense (i.e., meaning “must”). Similarly, the words“include”, “including”, and “includes” mean including, but not limitedto.

The following is a non-exhaustive list of embodiments which may beclaimed:

1. A method of determining volumetric measurement data of apredetermined anatomical feature, the method comprising:

monitoring a field of view of a depth sensing camera apparatus;identifying a predetermined anatomical feature as being present in thefield of view of the depth sensing camera apparatus;upon identification of the predetermined anatomical feature beingpresent in the field of view of the depth camera apparatus, determiningvolumetric measurement data of the identified anatomical feature presentin the field of view of the depth sensing camera apparatus; andoutputting the volumetric measurement data of the predeterminedanatomical feature.

2. The method of embodiment 1, the method comprising:

upon identification of the predetermined anatomical feature beingpresent in the field of view of the depth sensing camera apparatus,obtaining three-dimensional data of the predetermined anatomical featureand determining the volumetric measurement data of the predeterminedanatomical feature based on the three-dimensional data, wherein thethree-dimensional data comprises data acquired by the depth sensingcamera apparatus, or depth measurement or point cloud data generatedtherefrom.

3. The method of embodiment 1 or embodiment 2, comprising acquiring atwo-dimensional image from the depth sensing camera apparatus andidentifying the predetermined anatomical feature as being present in afield of view of the depth sensing camera apparatus in accordance withthe two-dimensional image.

4. A method of determining volumetric measurement data of apredetermined anatomical feature, the method comprising:

determining volumetric measurement data of one or more anatomicalfeatures present in the field of view of the depth sensing cameraapparatus;subsequently identifying one of the one or more anatomical features asbeing a predetermined anatomical feature; andoutputting the volumetric measurement data of the one of the one or moreanatomical features identified as the predetermined anatomical feature.

5. The method of embodiment 4, the method comprising:

obtaining three-dimensional data of the one or more anatomical featuresand determining the volumetric measurement data of the one or moreanatomical features based on the three-dimensional data, wherein thethree-dimensional data comprises data acquired by the depth sensingcamera apparatus, or depth measurement or point cloud data generatedtherefrom.

6. The method of embodiment 4 or embodiment 5, wherein the volumetricmeasurement data of one of the one or more features is associated withthe identified predetermined anatomical feature by comparing thevolumetric measurement data of each of the one or more anatomicalfeatures with a three-dimensional representation of a predeterminedanatomical feature, and identifying one of the one or more anatomicalfeatures as being the predetermined anatomical feature.

7. The method of embodiment 2 or embodiment 5, wherein determining thevolumetric measurement data comprises processing the three-dimensionaldata in dependence on a three-dimensional representation of thepredetermined anatomical feature.

8. The method of embodiment 7, wherein the three-dimensionalrepresentation of the predetermined anatomical feature is a deformablethree-dimensional model, and wherein the method comprises parametricallydeforming the three-dimensional model to fit the three-dimensional dataof the predetermined anatomical feature.

9. The method of embodiment 8, wherein the three-dimensional model isdeformed by one or more of rotation, altering a joint angle andvolumetric change.

10. The method of embodiment 7, wherein the three-dimensionalrepresentation of the predetermined anatomical feature is a deformablethree-dimensional model, and wherein the method comprises deforming thethree-dimensional data of the predetermined anatomical feature to fitthe three-dimensional model.

11. The method of any one embodiments 2, 5, and 7 to 10, wherein thethree-dimensional data comprises multiple data sets of anatomicalfeatures at different orientations.

12. The method of any one embodiments 2, 5, and 7 to 11, wherein data isperiodically or continuously received from the depth sensing cameraapparatus.

13. The method of any one embodiments 2, 5, and 7 to 12, wherein thethree-dimensional data comprises three-dimensional image data.

14. The method of any one embodiments 2, 5, and 7 to 13, wherein thethree-dimensional data represents at least one partial representation ofthe predetermined anatomical feature.

15. The method of any one embodiments 2, 5, and 7 to 14, comprisingrecording the time at which three-dimensional data is captured, andwherein the volumetric measurement data is generated in dependence onthe recorded time.

16. The method of any of the preceding embodiments, wherein thepredetermined anatomical feature is a limb, and is preferably a foot.

17. The method of any of the preceding embodiments, wherein the depthsensing camera apparatus comprises at least one emitter and one detectorarray, or at least two detector arrays.

18. A system comprising:

a depth sensing camera apparatus; anda processor coupled to the depth sensing camera apparatus;wherein the processor is configured to perform the method according toany one of embodiments 1 to 16.

19. The system of embodiment 17, wherein the depth sensing cameraapparatus comprises at least one emitter and one detector array, or atleast two detector arrays.

20. A computer readable medium having stored thereon instructions which,when executed on a processor, cause the processor to perform the methodaccording to any one of embodiments 1 to 16.

21. An method as substantially hereinbefore described with reference tothe accompanying figures.

22. An apparatus as substantially hereinbefore described with referenceto the accompanying figures.

1. A method of determining volumetric data of a predetermined anatomicalfeature, the method comprising: identifying one or more predeterminedanatomical features as being present in a field of view of athree-dimensional imaging camera apparatus; determining volumetric dataof the one or more predetermined anatomical features present in thefield of view of the three-dimensional imaging camera apparatus;associating the volumetric data of at least one of the one or morepredetermined anatomical features with the identified one or morepredetermined anatomical features; and outputting the volumetric data ofthe at least one of the said one or more predetermined anatomicalfeatures.
 2. The method of claim 1, wherein the step of determiningvolumetric data further comprises: estimating an outline of at least oneof the one or more predetermined anatomical features in one or moreimages.
 3. The method of claim 2, wherein the one or more images aretwo-dimensional images.
 4. The method of claim 2, wherein the outlinecomprises at least one partial outline.
 5. The method of claim 2,wherein the step of determining volumetric data further compriseseither: processing the outline using an estimate of a distance from areference point to the at least one of the one or more predeterminedanatomical features; or processing the outline using an estimate of adistance from a reference point to one or more points on the outline. 6.The method of claim 2, wherein the step of determining volumetric datafurther comprises: processing the outline using a three-dimensionalrepresentation of the at least one of the one or more predeterminedanatomical features.
 7. The method of claim 1, wherein the step ofdetermining volumetric data further comprises: utilizing a model thathas been produced based on at least one representation of the one ormore predetermined anatomical features.
 8. The method of claim 1,wherein the three-dimensional imaging camera apparatus captures at leasttwo spatial dimensions and a temporal dimension.
 9. The method of claim8, wherein the temporal dimension comprises a phase relationship betweenthe three-dimensional imaging camera apparatus and a light emittingapparatus.
 10. The method of claim 1, wherein the three-dimensionalimaging camera apparatus captures at least two spatial dimensions fromat least two viewpoints.
 11. The method of claim 1, wherein thethree-dimensional imaging camera apparatus captures at least two spatialdimensions in conjunction with a light emitting apparatus thatdifferentially emits light in at least one spatial dimension and thetemporal dimension, or in the at least two spatial dimensions.
 12. Themethod of claim 1, wherein the three-dimensional imaging cameraapparatus directly or indirectly captures at least three spatialdimensions.
 13. The method of claim 1, wherein the step of identifyingone or more predetermined anatomical features does not use an output ofthe three-dimensional imaging camera apparatus.